The use of Automatic Item Generation (AIG), the practice of creating assessment items algorithmically, is of increasing interest to educational assessors. AIG permits educational assessors to quickly generate assessment items and at least partially automate the generation of such assessment items. Furthermore, assessors can provide differing assessment items, each having a similar difficulty level, to improve test security. Moreover, adaptive testing having assessment items that vary systematically in difficulty can be produced. Such advantages have encouraged the research and development of AIG technologies.
However, AIG depends upon well-founded models of the cognitive abilities underlying performance. Where such models are lacking, AIG can only have heuristic usefulness. Conventional AIG research has been performed in areas where well-founded cognitive theories support the development of AIG algorithms, such as matrix completion and analytical reasoning. Such areas are generally restricted in content and highly structured.
In contrast, progress in AIG of verbal item types has been more limited, due to the openness of content and the considerable complexity of natural language. In open-ended verbal items, a strong preference exists for developing naturalistic items based upon actually published materials, and the most productive approaches have focused upon providing techniques to support test developers by supporting more efficient item selection and evaluation.
Where constrained item types have required natural language generation, the treatment of verbal materials has been straightforward and generally uses verbal templates to generate items. Typical template-based natural language generation includes two salient properties: 1) a list of phrases or sentences with open slots; and 2) the random or pseudo-random insertion of words from predetermined lists into particular slots. Template-based generation has the advantage of being straightforward, quick and dependent upon existing items. However, AIG from such simple templates is clearly limited because natural language complexities cannot be captured within a template format. Moreover, since the strings manipulated by template-based systems have no theoretical status, they do not support any principled analysis of the language employed in any particular problem type.
Conventional template-based AIG systems suffer from four distinct limitations: maintainability, output flexibility, output quality and an inability to easily produce multilingual outputs. In a template-based system, a large number of lists are stored and manipulated in order to generate textual output because each list is task or field specific. Accordingly, repetitive lists may be required to complete populate all task sets.
In addition, as the number of templates in a template-based system grows, it is more likely that the variety of templates disguises the systematic combination of a much smaller set of variables.
Moreover, systems must resolve context-dependencies inherent in language, such as subject-verb agreement, selection restriction (i.e., one drives a car, but flies an airplane), definite-indefinite selection (i.e., a student or the student), and the like. Such dependencies are handled ad hoc in a template-based system.
Finally, in order to produce a multilingual template-based system, a system maintainer must generate new templates for target language. Moreover, dependencies between templates and dependencies between entries in templates must be redefined for each template and/or combination of entries in the target language. As such, significant effort must be expended and significant resources must be dedicated to create a multilingual template-based system.
What is needed is a method and system for improving conventional automatic item generation by using non-template-based algorithms for generating assessment item text.
A need exists for a method and system for improving the maintenance of automatic item generation systems.
A further need exists for an automatic item generation system and method that increases output variety.
A still further need exists for an automatic item generation system and method that produces higher text quality.
A further need exists for a method and system of automatic test generation that more easily permits multilingual textual output.
The present invention is directed to solving one or more of the problems described above.